A logging framework is a software library or tool that provides a standardized way to record, format, filter, and route log messages generated by an application during its execution.
Logs below this level will be discarded
No logs yet
Emit a log message to see it here
Logging is critical for:
Well-designed frameworks abstract the complexity of logging so developers can focus on writing code without worrying about how logs are processed or where they go.
Frameworks like Log4j, SLF4J, and java.util.logging power millions of production services.
But what does it take to build such a system from scratch?
In this chapter, we will explore the low-level design of a logging framework in detail.
Lets start by clarifying the requirements:
Before starting the design, it's important to ask thoughtful questions to uncover hidden assumptions and clearly define the scope of the system.
Here is an example of how a discussion between the candidate and the interviewer might unfold:
Candidate: What log levels should the logging framework support?
Interviewer: It should support standard log levels like DEBUG, INFO, WARN, ERROR and FATAL. The system should filter messages based on a configured threshold (minimum log level).
Candidate: Do we need to support writing logs to multiple destinations?
Interviewer: Yes, the framework should support multiple destinations, including the console and file outputs.
Candidate: Can a single log message be sent to more than one destination simultaneously?
Interviewer: Yes, a log message can be routed to multiple output destinations at the same time.
Candidate: Do we need to support multiple log message formats ?
Interviewer: : Yes. Every log entry should include a timestamp and log level at a minimum. The system should support plugging in custom formatters.
Candidate: Will the framework be used in a multi-threaded environment? Do we need to ensure thread safety?
Interviewer: Yes, it will be used in a multi-threaded environment. The logging system must be thread-safe to ensure that log messages are not interleaved or lost.
Candidate: Should the framework support asynchronous logging to prevent blocking the main application thread?
Interviewer: That’s an important consideration. Yes, the framework should support asynchronous logging.
After gathering the details, we can summarize the key system requirements.
Core entities are the fundamental building blocks of our system. We identify them by analyzing the functional requirements and highlighting the key nouns and responsibilities that naturally map to object-oriented abstractions such as classes, enums, or interfaces.
Let’s walk through the functional requirements and extract the relevant entities:
This indicates the need for a LogLevel enum to represent the severity of each log message. It will be used to filter messages based on the configured minimum log level.
Additionally, we need a LogMessage class to encapsulate details of a single log entry such as the message content, timestamp, log level, thread name, and optional metadata.
The Logger class will serve as the primary interface exposed to client applications for logging messages.
This indicates the need for a LogAppender abstraction—an interface for defining output targets.
Concrete implementations might include:
ConsoleAppender: Writes logs to the standard output streamFileAppender: Writes logs to a specified fileThis implies the need for a LogFormatter abstraction to define how log messages are serialized before being passed to an appender.
Examples of formatter implementations include:
SimpleFormatter: Plain text logs with timestamp and levelJSONFormatter: Structured logs for machine parsing or log aggregation systemsThis requirement introduces the need for an AsyncLogProcessor or LogDispatcher component. It can use a background thread or queue to buffer log messages and dispatch them to appenders in a non-blocking manner.
This suggests a central LogManager entity responsible for creating and managing Logger instances.
Logger: The main interface clients use to log messages. Applies log level filtering and dispatches messages to configured appenders.LogLevel (Enum): Defines severity levels like DEBUG, INFO, WARN, ERROR, and FATAL.LogMessage: Represents a single log event. LogAppender (Interface): Abstraction for output destinations. LogFormatter: Defines how log messages are formatted.AsyncLogProcessor: Handles asynchronous logging.LogManager: Responsible for initializing and managing loggers.These entities define the key abstractions of a logging framework and provide a solid foundation for building a modular, extensible, and performant logging solution.
This section outlines the classes, interfaces, and their interactions, which form the architectural backbone of the logging framework.
The framework is composed of several types of classes, each with a distinct responsibility.
LogLevelA simple enumeration that defines the severity levels for log messages (DEBUG, INFO, WARN, ERROR, FATAL).
It includes a helper method, isGreaterOrEqual(), to easily compare severities and decide if a message should be logged.
LogMessageAn immutable data class (or DTO) that encapsulates all information about a single logging event.
It holds the timestamp, log level, logger name, thread name, and the actual message. This object is passed through the logging pipeline.
LoggerThe primary class that application developers interact with.
It provides methods like info(), warn(), etc., to trigger logging events. It manages its own log level and appenders, and it maintains a link to a parent Logger to form a hierarchy.
AsyncLogProcessorA background processor responsible for taking LogMessage objects and dispatching them to the appropriate appenders.
It uses a single-threaded executor to ensure logs are processed asynchronously without blocking the main application threads and are written in the correct order.
LogManagerA Singleton class that acts as the central point of configuration and management for the entire framework.
It is responsible for creating and managing the Logger hierarchy, holding the shared AsyncLogProcessor, and orchestrating a graceful shutdown.
The classes interact through a combination of composition, association, and implementation, creating a flexible and extensible system.
This "has-a" relationship implies ownership, where one object's lifecycle is managed by another.
LogManager has a AsyncLogProcessor and a map of all Logger instances. It creates and manages them.Logger has a list of LogAppenders.LogAppender (e.g., ConsoleAppender, FileAppender) has a LogFormatter. The formatter is an integral part of how the appender functions.This is a weaker "has-a" relationship where objects are related but have independent lifecycles.
Logger has a reference to its parent Logger, forming the core of the logger hierarchy. This is a self-referential association.This relationship exists where a concrete class provides a specific implementation for an interface.
SimpleTextFormatter implements the LogFormatter interface.ConsoleAppender and FileAppender implement the LogAppender interface.This relationship exists when one class uses another class.
Logger class depends on LogManager to get the global processor and creates LogMessage objects.AsyncLogProcessor depends on LogAppender and LogMessage to perform its work.Several design patterns are employed to ensure the framework is robust, flexible, and efficient.
This pattern is fundamental to the framework's flexibility.
The LogFormatter interface allows the algorithm for formatting a log message to be selected at runtime. We can easily add new formatters (e.g., JsonFormatter, XmlFormatter) and assign them to appenders without changing any other code.
The LogAppender interface allows the destination for log messages to be selected at runtime. A logger can be configured with multiple appenders to send logs to the console, a file, and a network endpoint simultaneously.
The LogManager.getLogger(name) method acts as a factory. It abstracts the creation of Logger instances, intelligently handling the construction of the logger hierarchy by parsing names and linking parents automatically.
The asynchronous logging mechanism is a classic example of this pattern. Application threads act as Producers, quickly creating LogMessage objects and submitting them. The dedicated AsyncLogProcessor thread acts as the sole Consumer, processing these messages from a queue, thereby decoupling the logging overhead from the application's performance.
This pattern is evident in the Logger hierarchy.
Logger checks its own level. If none is set, it delegates the request up to its parent, continuing up the chain to the root.additivity is enabled (the default), the event is passed up to the parent's appenders, forming a chain of processing.The Logger class itself serves as a simple facade. It provides a straightforward API (logger.info(...)) that hides the underlying complexity of message creation, level checks, asynchronous processing, formatting, and dispatching to appenders.
The LogManager is a singleton, providing a single, globally accessible point for managing loggers and the framework's lifecycle. This prevents conflicting configurations and ensures resources like the thread pool are shared.
LogLevel EnumDefines severity levels for log messages.
1class LogLevel(Enum):
2 DEBUG = 1
3 INFO = 2
4 WARN = 3
5 ERROR = 4
6 FATAL = 5
7
8 def is_greater_or_equal(self, other: 'LogLevel') -> bool:
9 return self.value >= other.valueThe isGreaterOrEqual() method helps determine whether a log message should be recorded based on the current logger’s configured level.
LogMessage ClassEncapsulates all details required for a log entry.
1class LogMessage:
2 def __init__(self, level: LogLevel, logger_name: str, message: str):
3 self.timestamp = datetime.now()
4 self.level = level
5 self.logger_name = logger_name
6 self.message = message
7 self.thread_name = threading.current_thread().name
8
9 def get_timestamp(self) -> datetime:
10 return self.timestamp
11
12 def get_level(self) -> LogLevel:
13 return self.level
14
15 def get_logger_name(self) -> str:
16 return self.logger_name
17
18 def get_thread_name(self) -> str:
19 return self.thread_name
20
21 def get_message(self) -> str:
22 return self.messageLogFormatter and Implementations (Strategy Pattern)To make the framework flexible, we use the Strategy Pattern to define interchangeable components for formatting logs and sending them to different destinations.
1class LogFormatter(ABC):
2 @abstractmethod
3 def format(self, log_message: LogMessage) -> str:
4 pass
5
6
7class SimpleTextFormatter(LogFormatter):
8 def format(self, log_message: LogMessage) -> str:
9 timestamp_str = log_message.get_timestamp().strftime("%Y-%m-%d %H:%M:%S.%f")[:-3]
10 return f"{timestamp_str} [{log_message.get_thread_name()}] {log_message.get_level().name} - {log_message.get_logger_name()}: {log_message.get_message()}\n"The SimpleTextFormatter formats logs into a human-readable format with timestamp, thread, level, logger, and message.
This design decouples the LogAppender (which handles output) from the specific format of the log message. We can easily create new formatters (e.g., JsonFormatter, XmlFormatter) and plug them into any appender without changing the appender's code.
LogAppender and ImplementationsStrategy interface for log destinations.
1class LogAppender(ABC):
2 @abstractmethod
3 def append(self, log_message: LogMessage):
4 pass
5
6 @abstractmethod
7 def close(self):
8 pass
9
10 @abstractmethod
11 def get_formatter(self) -> LogFormatter:
12 pass
13
14 @abstractmethod
15 def set_formatter(self, formatter: LogFormatter):
16 pass
17
18
19class ConsoleAppender(LogAppender):
20 def __init__(self):
21 self.formatter = SimpleTextFormatter()
22
23 def append(self, log_message: LogMessage):
24 print(self.formatter.format(log_message), end='')
25
26 def close(self):
27 pass
28
29 def set_formatter(self, formatter: LogFormatter):
30 self.formatter = formatter
31
32 def get_formatter(self) -> LogFormatter:
33 return self.formatter
34
35
36class FileAppender(LogAppender):
37 def __init__(self, file_path: str):
38 self.formatter = SimpleTextFormatter()
39 self._lock = threading.Lock()
40 try:
41 self.writer = open(file_path, 'a')
42 except Exception as e:
43 print(f"Failed to create writer for file logs, exception: {e}")
44 self.writer = None
45
46 def append(self, log_message: LogMessage):
47 with self._lock:
48 if self.writer:
49 try:
50 self.writer.write(self.formatter.format(log_message) + "\n")
51 self.writer.flush()
52 except Exception as e:
53 print(f"Failed to write logs to file, exception: {e}")
54
55 def close(self):
56 if self.writer:
57 try:
58 self.writer.close()
59 except Exception as e:
60 print(f"Failed to close logs file, exception: {e}")
61
62 def set_formatter(self, formatter: LogFormatter):
63 self.formatter = formatter
64
65 def get_formatter(self) -> LogFormatter:
66 return self.formatterThis design allows us to direct logs to various outputs (console, file, network socket, database) by creating new implementations of LogAppender. A logger can be configured with multiple appenders to send the same log message to several destinations simultaneously.
FileAppender writes logs to a file, handling synchronization and fallback gracefully. Useful for persistent logging in production systems.
To minimize the performance impact on the main application threads, log processing is handled asynchronously.
1class AsyncLogProcessor:
2 def __init__(self):
3 self.executor = ThreadPoolExecutor(max_workers=1, thread_name_prefix="AsyncLogProcessor")
4 self.shutdown_flag = False
5
6 def process(self, log_message: LogMessage, appenders: List[LogAppender]):
7 if self.shutdown_flag:
8 print("Logger is shut down. Cannot process log message.", file=sys.stderr)
9 return
10
11 def process_task():
12 for appender in appenders:
13 appender.append(log_message)
14
15 self.executor.submit(process_task)
16
17 def stop(self):
18 self.shutdown_flag = True
19 self.executor.shutdown(wait=True, timeout=2)
20 if not self.executor._shutdown:
21 print("Logger executor did not terminate in the specified time.", file=sys.stderr)This class uses a dedicated background thread to format and append log messages, freeing the application thread to continue its work immediately after a log call.
The application threads act as "producers," submitting LogTask objects to a BlockingQueue. The AsyncLogProcessor runs a single "consumer" thread that takes tasks from the queue and processes them.
Using a SingleThreadExecutor ensures that logs are written in the order they were submitted, which is a critical requirement for a logging framework.
Acts as the centralized manager for loggers.
1class LogManager:
2 _instance = None
3 _lock = threading.Lock()
4
5 def __init__(self):
6 if LogManager._instance is not None:
7 raise Exception("This class is a singleton!")
8 self.loggers: Dict[str, 'Logger'] = {}
9 self.root_logger = Logger("root", None)
10 self.loggers["root"] = self.root_logger
11 self.processor = AsyncLogProcessor()
12
13 @staticmethod
14 def get_instance():
15 if LogManager._instance is None:
16 with LogManager._lock:
17 if LogManager._instance is None:
18 LogManager._instance = LogManager()
19 return LogManager._instance
20
21 def get_logger(self, name: str) -> 'Logger':
22 if name not in self.loggers:
23 self.loggers[name] = self._create_logger(name)
24 return self.loggers[name]
25
26 def _create_logger(self, name: str) -> 'Logger':
27 if name == "root":
28 return self.root_logger
29
30 last_dot = name.rfind('.')
31 parent_name = "root" if last_dot == -1 else name[:last_dot]
32 parent = self.get_logger(parent_name)
33 return Logger(name, parent)
34
35 def get_root_logger(self) -> 'Logger':
36 return self.root_logger
37
38 def get_processor(self) -> AsyncLogProcessor:
39 return self.processor
40
41 def shutdown(self):
42 # Stop the processor first to ensure all logs are written
43 self.processor.stop()
44
45 # Then, close all appenders
46 all_appenders = set()
47 for logger in self.loggers.values():
48 for appender in logger.get_appenders():
49 all_appenders.add(appender)
50
51 for appender in all_appenders:
52 appender.close()
53
54 print("Logging framework shut down gracefully.")Handles:
The main interface for developers.
1class Logger:
2 def __init__(self, name: str, parent: Optional['Logger']):
3 self.name = name
4 self.level: Optional[LogLevel] = None
5 self.parent = parent
6 self.appenders: List[LogAppender] = []
7 self.additivity = True
8
9 def add_appender(self, appender: LogAppender):
10 self.appenders.append(appender)
11
12 def get_appenders(self) -> List[LogAppender]:
13 return self.appenders
14
15 def set_level(self, min_level: LogLevel):
16 self.level = min_level
17
18 def set_additivity(self, additivity: bool):
19 self.additivity = additivity
20
21 def get_effective_level(self) -> LogLevel:
22 logger = self
23 while logger is not None:
24 current_level = logger.level
25 if current_level is not None:
26 return current_level
27 logger = logger.parent
28 return LogLevel.DEBUG # Default root level
29
30 def log(self, message_level: LogLevel, message: str):
31 if message_level.is_greater_or_equal(self.get_effective_level()):
32 log_message = LogMessage(message_level, self.name, message)
33 self._call_appenders(log_message)
34
35 def _call_appenders(self, log_message: LogMessage):
36 if self.appenders:
37 LogManager.get_instance().get_processor().process(log_message, self.appenders)
38
39 if self.additivity and self.parent is not None:
40 self.parent._call_appenders(log_message)
41
42 def debug(self, message: str):
43 self.log(LogLevel.DEBUG, message)
44
45 def info(self, message: str):
46 self.log(LogLevel.INFO, message)
47
48 def warn(self, message: str):
49 self.log(LogLevel.WARN, message)
50
51 def error(self, message: str):
52 self.log(LogLevel.ERROR, message)
53
54 def fatal(self, message: str):
55 self.log(LogLevel.FATAL, message)Each logger:
LogManagerThe additivity flag controls whether messages should propagate up the logger hierarchy.
The LoggingFrameworkDemo class demonstrates how a client would configure and use the framework.
1class LoggingFrameworkDemo:
2 @staticmethod
3 def main():
4 # --- 1. Initial Configuration ---
5 log_manager = LogManager.get_instance()
6 root_logger = log_manager.get_root_logger()
7 root_logger.set_level(LogLevel.INFO) # Set global minimum level to INFO
8
9 # Add a console appender to the root logger
10 root_logger.add_appender(ConsoleAppender())
11
12 print("--- Initial Logging Demo ---")
13 main_logger = log_manager.get_logger("com.example.Main")
14 main_logger.info("Application starting up.")
15 main_logger.debug("This is a debug message, it should NOT appear.") # Below root level
16 main_logger.warn("This is a warning message.")
17
18 # --- 2. Hierarchy and Additivity Demo ---
19 print("\n--- Logger Hierarchy Demo ---")
20 db_logger = log_manager.get_logger("com.example.db")
21 # db_logger inherits level and appenders from root
22 db_logger.info("Database connection pool initializing.")
23
24 # Let's create a more specific logger and override its level
25 service_logger = log_manager.get_logger("com.example.service.UserService")
26 service_logger.set_level(LogLevel.DEBUG) # More verbose logging for this specific service
27 service_logger.info("User service starting.")
28 service_logger.debug("This debug message SHOULD now appear for the service logger.")
29
30 # --- 3. Dynamic Configuration Change ---
31 print("\n--- Dynamic Configuration Demo ---")
32 print("Changing root log level to DEBUG...")
33 root_logger.set_level(LogLevel.DEBUG)
34 main_logger.debug("This debug message should now be visible.")
35
36 try:
37 time.sleep(0.5)
38 log_manager.shutdown()
39 except Exception as e:
40 print("Caught exception")
41
42if __name__ == "__main__":
43 LoggingFrameworkDemo.main()The driver code showcases the framework's key features:
Which component in a logging framework is responsible for sending log messages to a specific destination such as a file or console?
No comments yet. Be the first to comment!